Linear Manifold Correlation Clustering

The detection of correlations is a data mining task of increasing importance due to new areas of application such as DNA microarray analysis, collaborative filtering, and text mining. In these cases object similarity is no longer measured by physical distance, but rather by the behavior patterns objects manifest or the magnitude of correlations they induce. Many approaches have been proposed to identify clusters complying with this requirement. However, most approaches assume specific cluster models, which in turn may lead to biased results. In this paper we present a novel methodology based on linear manifolds which provides a more general and flexible framework by which correlation clustering can be done. We discuss two stochastic linear manifold cluster models and demonstrate their applicability to a wide range of correlation clustering situations. The general model provides the ability to capture arbitrarily complex linear dependencies or correlations. The specialized model focuses on simpler forms of linear dependencies, yet generalizes the dependencies often sought by the so called “pattern” clustering methods. Based on these models we discuss two linear manifold clustering algorithms, the later a fine-tuned derivative of the first targeting simpler forms of correlation and “pattern” clusters. The efficacy of our methods is demonstrated by a series of experiments on real data from the microarray and collaborative filtering domains. One of the experiments demonstrates that our method is able to identify statistically significant correlation clusters that are overlooked by existing methods.